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The  Army  is  committed  to  exploiting  the  benefits  of  modeling  and  simulation  (M&S)  to  improve 
the  acquisition  process.  M&S  is  expected  to  simultaneously  support  the  goals  of  “faster,  better 
and  cheaper.”  M&S  will  enable  continual  involvement  of  the  user  in  requirements  evolution,  will 
reduce  cycle-time  by  means  of  more  efficient  development  processes,  and  will  lower  the 
ultimate  cost  of  ownership  by  addressing  all  aspects  of  sustainability  early  in  a  program.  Army 
interest  is  motivated  and  sustained  by  the  ever-increasing  availability  of  enabling  technologies 
for  M&S,  by  examples  of  successful  implementations  of  virtual  prototyping  in  commercial 
industry,  and  by  impressive  displays  of  virtual-reality  and  special  effects  by  the  entertainment 
industry.  These  developments  suggest  that  a  bold  new  strategy  for  virtual  design  and 
development  is  “on  the  horizon.”  However,  models  and  simulations  are  inherently  limited  in 
what  they  can  do,  i.e.,  by  their  very  nature  they  only  approximate  reality.  Moreover,  the 
fundamental  management  constraint  embedded  in  simultaneously  addressing  cost,  schedule 
and  performance,  i.e.,  a  program  manager  can  reasonably  set  only  two  of  the  three,  suggests 
there  are  limits  to  what  can  be  accomplished.  For  all  these  reasons,  M&S  should  be  an 
essential  tool  for  a  PM  to  improve  his  program,  but  it  should  not  be  expected  to  be  a  panacea 
for  all  program  challenges  nor  should  a  single  M&S  approach  be  expected  to  fit  all  program 
needs.  The  purpose  of  this  research  is  to  determine  reasonable  expectations  of  potential  M&S 
contributions  to  the  acquisition  process  and  to  propose  focus  areas  for  the  Army  M&S 
community  in  order  to  maximize  the  benefits  to  the  acquisition  community. 
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CALIBRATING  EXPECTATIONS:  THE  BENEFITS  AND  THE  LIMITATIONS  OF  MODELS  AND  SIMULATIONS  IN 

THE  ACQUISITION  PROCESS 


INTRODUCTION 

There  is  a  tide  in  the  affairs  of  men,  which,  taken  at  the  flood,  leads  on  to  fortune; 

omitted,  all  the  voyage  of  their  life  is  bound  in  shallows  and  in  miseries 

— -William  Shakespeare 

The  great  bard’s  view  of  lost  opportunity  from  his  play  Julius  Caesar  is  almost  as 
appropriate  to  the  subject  of  this  paper  as  is  the  old  adage,  “Opportunity  knocks  but  once,”  with 
one  important  distinction.  Even  though  the  Army  is  still  pursuing  a  simulation-based  approach 
to  acquisition,  it  still  has  the  opportunity  to  exploit  models  and  simulations  (M&S)  in  the 
acquisition  process  just  as  the  visionaries  of  the  early  1 990’s  described.  The  potential  benefits 
of  M&S  are  forever  nearing  Shakespeare’s  “flood  stage”,  and  the  opportunity  to  exploit  them  will 
continue  to  knock;  it’s  only  a  question  of  when  or  if  we’ll  open  the  door. 

EARLY  EXPECTATIONS 

Interest  in  simulation  exploded  in  the  late  1980’s  and  was  driven  in  part  by  the  concept 
of  distributed  simulation  as  embodied  in  the  Defense  Advanced  Research  Projects  Agency 
(DARPA)  Simulation  Networking  (SIMNET)  program.  In  the  truest  sense  of  the  phrase, 
SIMNET  enabled  a  paradigm  shift  in  how  training  could  be  conducted.  Instead  of  assembling 
soldiers  for  live  training,  SIMNET  enabled  troops  in  geographically-dispersed  units,  to  use 
manned  simulators  in  dynamic,  free-play  exercises  on  a  virtual  battlefield.  Moreover,  The  visual 
nature  of  SIMNET  captured  the  imaginations  of  many  inventive  people  and  stimulated  a  host  of 
innovative  applications. 


The  SIMNET-inspired,  distributed-simulation  paradigm  seemed  to  be  all  things  to  all 
people  at  the  same  time.  The  vision  depicted  in  Figure  1  was  perhaps  the  most  elegant 
description  of  the  ultimate  end-state  if  the  concept  were  carried  to  its  natural  conclusion.1  It 
shows  individual,  discipline-specific  M&S  capabilities  -  each  of  which  are  valuable  tools  in  their 


FIGURE  1  SIMULATION  VISION 


own  right  -  interconnected  via  the  internet  to 
achieve  far  greater  effect  than  each  could 
alone.  In  the  training  arena,  distributed 
simulation  held  forth  the  vision  of  large-scale 
training  exercises  that  were  lower  cost,  safer 
and  which  could  be  conducted  without 
damage  to  the  environment.  Simulators  were 
to  be  inexpensive  enough,  i.e.,  less  than 
$250k,  that  SIMNET  sites  could  be  populated 
throughout  the  active  and  reserve 
components.  In  the  operational  world, 
distributed  simulation  was  to  enable  planners 
to  rapidly  generate  terrain  databases  of  the 


latest  trouble  spot,  to  conduct  mission  rehearsals,  and  to  evaluate  proposed  courses  of  action. 
Soldiers  could  rehearse  their  missions  en  route  and  in  staging  areas  prior  to  the  start  of  their 


mission.  The  simulation  would  be  all  the  more  realistic  because  the  computer-generated 


opponents  would  be  programmed  to  fight  with  the  tactics  of  the  expected  adversary.  Even 
better,  the  threat  would  be  “dialable”,  that  is,  it  could  behave  like  Soviet  forces  one  moment  and 
Iraqi  troops  the  next.  Developers,  who  previously  promoted  “fly  before  you  buy”  so  that  users 
could  evaluate  products  in  hardware  prototypes  before  purchasing  them,  now  proposed  to 
employ  distributed  simulation  to  provide  users  with  yet  another  level  of  risk  reduction.  Users 
could  simulate  before  they  ’’flew”  by  shaking  out  concepts  on  the  virtual  battlefield  before  metal 
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was  even  bent  on  a  prototype.  Real  visionaries  spoke  of  true  “virtual  prototyping,”  a  process  in 
which  all  development  was  performed  in  simulation  and  the  very  first  instance  in  hardware 
would  be  a  production  item  ready  for  fielding.  At  nearly  the  same  time,  commercial  Industry 
made  great  strides  in  applying  virtual  prototyping  techniques  to  substantially  improve  their 
design  and  development  processes.  Most  notable  was  the  Boeing  Corporation  development  of 
the  Boeing  ”777”  airplane  which  was  done  almost  entirely  by  using  computer  models  and 
representations. 

There  was  a  host  of  reasons  to  expect  that  a  new  approach  to  training  and  equipping  the 
force  could  be  facilitated  by  distributed  simulation.  While  the  computer  capabilities  of  the  time 
did  limit  performance,  there  seemed  to  be  no  technical  obstacles  to  achieving  the  goal  of  a  more 
efficient  and  a  more  cost-effective  procurement  process.  Admittedly,  the  simulation  technology 
incorporated  in  early  SIMNET  was  crude,  especially  by  today’s  standards,  but  it  was  remarkably 
effective  and  was  improving  all  the  time.  Moores’  Law,2  “Every  eighteen  months,  processing 
power  doubles  while  costs  hold  constant,”  legitimized  the  belief  that  technological  immaturity 
was  only  a  near-term  barrier  to  success.  Command  emphasis  on  the  use  of  M&S  from  the 
Chief  of  Staff,  Army,  downward  was  rampant.  Simulation  Master  Plans  became  part  of  a 
program  manager’s  requisite  planning  process.  Defense  and  Service  simulation  offices  were 
created.  Engineers  and  scientists  demonstrated  natural  extensions  of  the  philosophy  of 
concurrent  engineering  by  interconnecting  engineering  design  tools  in  a  distributed-simulation 
paradigm3  and  by  analyzing  design  concepts  with  an  operator  directly  in  the  design  loop4. 
Metcalfe’s  Law,  “The  utility  of  a  network  increases  with  the  square  of  the  number  of  users,”5 
hinted  at  the  cumulative  synergistic  effect  of  widespread  collaboration  in  distributed  simulation. 
Resources  were  available.  Commercial  industry  proved  that  it  could  be  done;  everyone  just 
knew  the  Army  could  do  it  too.  The  only  open  question  was  just  how  successful  the  Army  would 
ultimately  be.  Yet  a  decade  later,  the  vision  is  still  unfulfilled. 
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EARLY  CHALLENGES  AND  MISSTEPS 


A  candid  assessment  points  to  a  number  of  reasons  for  the  Army’s  failure  to  achieve  the 

goal. 

Exaggerated  schedule  and  performance  claims  by  supporters  about  M&S  capability  to 
do  everything  “faster,  better  and  cheaper”  eroded  confidence.  Unreasonably  inflated 
expectations  of  M&S  customers,  many  of  which  were  not  met,  caused  resentment  in  the  field 
and  a  reassessment  of  what  was  really  doable  in  the  near-term.  This  author  can  testify  from 
personal  experience  as  the  final  SIMNET  Program  Manager  about  the  extreme  disconnect 
between  user  expectations  and  what  was  actually  deliverable  under  the  SIMNET  contracts. 

Furthermore,  concerns  about  the  inherent  limitations  of  M&S  as  well  as  the  inappropriate 
use  and  the  potential  abuse  of  M&S  in  analyses  heightened  the  awareness  of  the  operations- 
research  community.  These  concerns  as  well  as  the  test  community’s  objection  to  testing  in 
simulation  instead  of  hardware  served  to  drive  M&S  progress  to  a  more  measured  pace. 

Declining  resources,  during  the  draw-down  of  forces  in  the  early  1990’s,  coupled  with  an 
unrealistically  optimistic  estimate  of  the  true  cost  of  implementing  almost  all  parts  of  the  vision, 
caused  a  reassessment  of  M&S  priority  in  the  budget  process.  Simultaneously,  cost-reduction 
initiatives  began  to  shift  the  focus  of  M&S  into  a  cost-savings  tool. 

Finally,  in  this  author’s  view,  the  most  important  reason  that  a  simulation-based 
procurement  process  is  still  “on  the  horizon”  is  that  the  Army  lacked  an  overarching  vision  which 
could  focus  the  contributions  of  all  M&S  participants  toward  a  common  goal.  The  Army  and 
Department  of  Defense  vision  statements6, 7  speak  broadly  in  terms  of  interoperability  of  models, 
of  open  architecture,  and  of  integrating  M&S  tools  into  the  acquisition  process.  However,  the 
bulk  of  acquisition  simulation  is  performed  within  individual  program  offices,  and  such  high-level 
goals  are  beyond  the  resources  and  the  charters  of  most  program  managers.  As  a  result,  M&S 
literature  is  rife  with  success  stories  that  were  program-specific8, 9' 10, 11  but  which  were  not 
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generally  applicable  to  the  wider  user  community.  A  lot  of  good  M&S  work,  by  intelligent  and 
dedicated  professionals,  advanced  individual  program-office  goals  but  was  so 
compartmentalized  that  the  corporate  Army  and  the  Defense  Department  was  unable  to  exploit 
the  synergy  of  their  efforts.  Neither  was  the  Army  able  to  formally  incorporate  the  individual 
program-office  M&S  efforts  into  the  acquisition  process. 

The  puzzling  thing  is  that  the  various  visions  proposed  by  the  trainers,  operators  and 
developers  are  still  as  desirable  today  as  they  were  when  they  were  first  surfaced  10  years  ago; 
it  would  be  difficult  to  state  them  any  better.  Moreover,  technology  has  advanced  as  predicted 
and  techniques  have  improved  accordingly.  Failure  in  the  face  of  such  progress  begs  the 
question,  “Why  aren’t  we  there  yet?” 

LIMITATIONS 

Analysts  write  about  war  as  if  it’s  choreographed  ahead  of  time,  and  when  the 
orchestra  strikes  up  and  starts  playing,  everyone  goes  out  and  plays  a  set  piece. 

What  I  say  to  these  folks  is,  “Yes,  it’s  choreographed,  and  what  happens  is  the 
orchestra  starts  playing,  and  some  SOB  climbs  out  of  the  orchestra  pit  with  a 
bayonet  and  starts  chasing  you  around  the  stage,  and  the  choreography  goes 
right  out  the  window.” 

— GEN  Norman  Schwartzkopf 

The  Commanding  General  of  Allied  Forces  during  Operation  Desert  Storm  made  this 
comment  at  a  press  briefing  in  response  to  a  reporter’s  question  about  the  conduct  of  the 
operation.  GEN  Scwartzkopf’s  answer  vividly  captures  the  essence  of  the  fog  of  war  and  points 
to  perhaps  the  most  fundamental  challenge  for  military  planners  -  the  inherent  unpredictability 
of  warfare.  When  one  also  considers  the  approximate  nature  of  even  the  best  models,  the 
challenge  to  the  M&S  community  is  even  more  daunting. 

UNKNOWNS  AND  UNKNOWN  UNKNOWNS 


5 


All  models  are  approximations  to  reality  and  have  limited  ranges  of  validity.  Dr  Solomon 
W.  Golomb,  Professor  of  Communications  at  the  University  of  Southern  California,  summarized 
limitations  and  pitfalls  of  using  models  and  simulations  in  the  form  of  ‘Ten  Commandments”  as 
shown  in  Table  I.12 


1  Thou  shalt  not  believe  the  33rd  order 
consequences  of  a  first  order  model. 

VI  Thou  shalt  not  limit  thyself  to  a  single  model 
-  more  than  one  [model]  may  be  useful  for 
understanding  different  aspects  of  the  same 
phenomenon. 

II  Thou  shalt  not  extrapolate  beyond  the 
regions  of  fit. 

VII  Thou  shalt  not  retain  a  discredited  model. 

Ill  Thou  shalt  not  apply  any  model  until  you 
understand  the  simplifying  assumptions  on 
which  it  is  based  and  can  test  their 
applicability. 

VIII  Thou  shalt  not  fall  in  love  with  thy  model. 

IV  Thou  shalt  not  believe  that  the  model  is 
reality. 

IX  Thou  shalt  not  apply  the  terminology  of 
Subject  A  to  the  problems  of  Subject  B  if  it  is 
to  the  enrichment  of  neither. 

V  Thou  shalt  not  distort  reality  to  fit  the  model. 

X  Thou  shalt  not  expect  that  having  named  a 
demon  thou  hast  destroyed  him. 

TABLE  1  TEN  COMMANDMENTS  OF  MODELS  AND  SIMULATIONS 


Analysts  familiar  with  M&S  literature,  especially  those  who  maintain  contacts  through  the 
various  professional  M&S  conferences,  have  most  likely  seen  similar  expressions  of  guidance 
on  the  use  of  models.  For  the  most  part,  these  guidelines  are  self-explanatory,  and  like 
Murphy’s  proverbial  laws,  one  can  often  relate  them  to  his  personal  experience.  Insofar  as  this 
author  has  been  able  to  determine,  Dr  Golomb’s  final  two  commandments  are  unique  to  him  but 
make  a  valuable  point.  With  respect  to  commandment  number  9,  Dr  Golomb  adds,  ‘The 
purpose  of  terminology  and  notation  should  be  to  enhance  insight  and  facilitate  computation  — 
not  to  impress  the  uninitiated  with  gobbledygook.”13  Together  with  the  final  commandment,  it 
serves  as  both  a  warning  to  the  modeler  and  as  counsel  to  decision-makers;  the  burden  of 
explanation  of  the  results,  method  and  assumptions  rests  with  the  modeler. 


6 


Unknowns,  or  as  they  are  sometimes  referenced  “unks,”  are  the  bane  of  the  modeler. 
Modelers  must  often  estimate  the  effects  of  variables  they  cannot  otherwise  accurately  measure 
or  characterize.  Examples  include  the  effects  of  degraded  performance  caused  by  fatigue  and 
by  stress.  Coincidentally,  both  are  good  examples  of  Dr.  Golomb’s  10th  Commandment;  having 
named  a  section  of  code  “degraded  performance”  does  not  necessarily  mean  that  true  effect 
has  been  captured.  As  troubling  as  unknowns  may  be,  even  more  troubling  are  the  unknown 
unknowns  or  “unk  unks,”  those  variables  which  affect  an  analysis  but  of  which  the  modeler  is 
completely  unaware.  “Unk  unks”  often  surface  when  subsystem  models  are  combined  to 
represent  an  aggregate  system.  Interrelationships  among  subsystems,  which  had  not  been 
anticipated  by  the  individual  subsystem-modelers,  become  painfully  apparent  when  the  overall 
system-model  behaves  inappropriately.  Usually,  this  is  accepted  as  a  cost  of  doing  business, 
and  modelers  resign  themselves  to  the  fact,  “You  don’t  know  what  you  don’t  know.”  Bran 
Ferren,  President  for  Research  and  Development,  Creative  Technology  at  Walt  Disney 
Imagineering  extended  this  line  of  reasoning  to  astonishing  effect.  He  observed  that  there  are 
cases  of  commonly-used  applications  which  cannot  be  and  most  likely  will  never  be  understood 
by  any  one  person.14  He  cited  the  Windows™  operating  system  for  personal  computers  as  an 
example.  Teams  of  individuals  may  be  identified  who  collectively  possess  the  understanding  of 
how  Windows™  operates,  but  no  single  individual  is  capable  of  comprehending  all  aspects  of 
the  software.  Extending  the  modeler’s  assessment  of  “unk  unks,”  one  must  conclude,  “Not  only 
do  we  not  know  what  we  don’t  know,  we  don’t  even  know  what  we  do  know.” 15 

Dr.  LaBerge,  a  former  Under  Secretary  of  the  Army,  suggests  one  key  way  to 
substantially  improve  the  capability  for  M&S  to  support  the  decision-making  process  is  to 
implement  peer  review  of  M&S  results.16  He  convincingly  argues  that  the  goals  of  verification, 
validation  and  accreditation  (W&A)  are  better  served  by  the  peer-review  process  of  scientific 
research  practiced  in  the  academic  environment.  Just  as  articles  submitted  for  publication  in 
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refereed  journals  undergo  peer  review,  so  should  the  results  of  modeling  and  simulation. 
Decision-makers  cannot  be  expected  to  verify  correctness  or  applicability  of  M&S  claims,  and 
the  current  W&A  process  only  authenticates  methodology,  not  results.  This  recommendation  is 
especially  attractive  because  of  its  potential  for  better-informed  decisions  and  because  of  the 
ease  with  which  it  could  be  implemented  within  the  framework  of  networked  M&S  experts 
depicted  in  Figure  1 .  Peer  review  would  also  address  concerns  that  the  current  VV&A  process 
is  expensive  and  not  well  understood17  and  help  to  reduce  the  number  of  unknowns  in  the 
analysis. 

OUTSIDE  INFLUENCES 

No  discussion  of  the  limitations  of  M&S  would  be  complete  without  recognizing  that 
outside  influences,  such  as  emotion,  mistrust,  and  hidden  agendas,  can  adversely  influence 
decisions  even  if  a  simulation  is  well  planned  and  well  run. 

Consider  the  circumstances  surrounding  the  simulation  of  the  final  day  of  the  siege  of 
the  Branch  Davidian  compound  by  FBI  agents  in  Waco  in  April,  1993.18  At  issue  is "...  whether 
the  FBI  is  being  truthful  about  its  role  in  the  standoff  at  the  Branch  Davidians’  Mount  Carmel 
compound  before  it  burned  to  the  ground  on  April  19, 1993.”19  Lawyers  representing  survivors 
of  the  siege  in  a  wrongful-death  suit  against  the  Government  maintain  that  the  FBI  fired 
weapons  into  the  compound  before  it  erupted  into  flames  and  cite  as  evidence  repeated  flashes 
recorded  by  airborne,  infrared  cameras  which  were  flown  by  the  FBI  over  the  compound  that 
fateful  day.  The  FBI  denies  that  its  agents  shot  into  the  compound  on  the  day  of  the  fire.  A 
reenactment  at  FT  Hood  was  intended  to  answer  any  lingering  doubts.  Two  aircraft  using  the 
same  cameras  as  were  employed  during  the  actual  siege  photographed  a  team  of  soldiers  and 
postal  inspectors  as  they  executed  a  series  of  drills  and  fired  weapons.  Analysts  expected  that 
the  film  taken  during  the  reenactment  would  indicate  whether  the  unexplained  flashes  cited  by 
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the  plaintiffs’  lawyers  were  the  result  of  FBI  gunfire  or  could  be  explained  by  other  factors  such 
as  reflections  from  surface  water  or  other  debris.  Unfortunately  for  those  hoping  for  a  speedy 
explanation,  the  videos  have  been  sealed  instead  of  being  released  to  the  public.  This  action, 
taken  by  a  U.S.  District  Court  judge,  led  at  least  one  reporter  to  observe,  ‘The  lack  of  public 
access  creates  the  possibility  that  both  sides  in  the  case  will  release  conflicting  opinions  without 
any  public  viewing  of  the  videos.”  True  to  the  reporter’s  prediction,  on  the  very  next  day, 
lawyers  for  both  the  Government  and  the  plaintiffs  claimed  that  the  videotape  conclusively 
supported  their  respective  positions.20  Four  days  later,  both  parties  acknowledged  that  better 
copies  of  the  videotaped  reenactment  would  be  needed  before  their  experts  could  make  a 
determination.21 

It  is  not  my  intent  to  judge  the  actions  of  the  parties  involved;  this  case  is  cited  because 
of  its  national  interest  and  to  highlight  several  complicating  factors.  The  plaintiffs’  case  was 
motivated  by  mistrust  of  the  Government’s  account  of  events  during  the  siege.  Under  the 
circumstances,  it’s  unlikely  that  the  results  of  the  simulation  would  be  accepted  unless  they 
conclusively  answered  the  question  or  at  least  supported  the  plaintiffs’  position.  Considering  the 
conflicting  claims  made  by  experts  viewing  the  same  data,  the  only  thing  one  can  safely 
conclude  is  that  at  least  half  of  the  experts  are  wrong.  The  judge’s  actions,  to  first  close  the 
simulation  to  key  personnel  and  then  to  seal  the  results,  highlights  the  importance  of  concerns 
beside  just  getting  to  the  facts  of  the  case.  No  matter  how  carefully  the  reenactment  was 
planned  and  executed,  the  usefulness  of  the  simulation  in  such  an  atmosphere  of  mistrust  was 
doomed  to  failure. 

Now  consider  the  rationale  behind  the  preference  of  the  F-22  Program  Office  for  using 
modeling  and  simulation  instead  of  hardware  tests  to  predict  the  performance  of  its  aircraft. 
“Modeling  and  Simulation  offers  the  F-22  program  another  benefit,  Air  Force  sources  said, 
because  the  Service  would  control  the  inputs  into  the  model,  the  outcome  -  proving  the 
aircraft’s  effectiveness  -  is  much  easier  to  shape  than  the  outcome  of  an  open  air  test  with  any 
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number  of  unanticipated  variables.”22  Clearly  insightful  use  of  M&S  allows  one  to  get  the 
answer  he  “knows  is  right.”  Is  it  any  wonder  the  test  community  strenuously  mistrusts  and 
opposes  the  practice  of  test  and  evaluation  in  simulation?  “We  have  no  evidence  to  show  us 
that  you  can  substitute  modeling  and  simulation  in  the  live  fire  world  for  live  fire  tests.”23 

Lack  of  trust  among  acquisition  professionals  was  also  one  of  the  issues  cited  in  an  early 
assessment  of  M&S  in  acquisition.24  Specifically,  the  study  recognized  that  program  offices 
overly  control  access  to  their  system  models  to  prevent  their  misuse,  i.e.,  evidence  of  mistrust  of 
those  outside  the  program.  The  study  also  cited  program  office  concerns  that  their  programs 
might  be  subject  to  side-by-side  comparisons  and  trade-off  analyses  with  competing  programs 
and  hence  an  unwillingness  to  participate  if  the  results  were  unlikely  to  be  favorable. 


POTENTIAL 

The  more  you  know,  the  harder  it  is  to  take  decisive  action.  Once  you  become 
informed,  you  begin  seeing  complexity  and  shades  of  gray.  You  realize  that 
nothing  is  as  simple  as  it  first  appears.  Ultimately,  knowledge  is  paralyzing. 
Being  a  man  of  action,  I  can’t  afford  to  take  that  risk. 

— Calvin  [of  Calvin  &  Hobbes] 


PERFORMANCE 

One  “commonly  known  fact”  is  that  how  well  a  system  is  designed  will  determine  how 
much  it  will  cost  to  sustain  it.  One  source  on  simulation  for  acquisition  quantifies  the  amount  in 
the  terms,  “70  to  90  percent  of  [a  system’s]  life  cycle  funds  are  committed  in  the  first  10  to  30 
percent  of  the  life  cycle.”25  If  this  is  true,  and  it  generally  is  accepted  as  such,  then  M&S  efforts 
must  focus  on  the  early  design  and  development  stage  of  a  program  if  it  is  to  produce  the 
greatest  cost  and  performance  benefits.  The  practice  best  suited  to  promote  this  early 
involvement  is  concurrent  engineering,  “a  systematic  approach  to  the  integrated,  concurrent 
design  of  products  and  their  related  processes,  including  manufacture  and  support ... 
considering  all  elements  of  the  product  life  cycle  from  conception  through  disposal,  including 
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quality,  cost,  schedule  and  requirements.”26  M&S  promotes  collaboration  among  discipline- 
specific  experts  and  facilitates  the  exchange  of  design  information  among  a  multi-disciplinary 
team. 

In  an  often-cited 
reference  for  automotive 
industry,  The  Machine  That 
Changed  The  World:  The 
Story  of  Lean  Production, 
the  authors  argue  that  the 
four  elements  of  lean 
design  are  leadership, 
teamwork,  communication, 
and  simultaneous 
development.27  These 

elements  are  equally  appropriate  in  the  context  of  military  acquisitions,  and  M&S  is  the  ideal 
means  to  facilitate  all  aspects  of  the  process.  The  authors  maintain  that  collaboration  among  a 
multidisciplinary  design  team,  in  which  ideas  are  openly  exchanged  and  resolved  at  the  team 
level,  and  specific  consideration  of  all  aspects  of  the  production  and  fielding  process  are 
essential  for  efficient  product  development.  Intuitively,  their  argument  makes  sense  and  even 
though  they  avoid  many  of  the  commonly  used  terms,  their  statement  captures  the  essence  of 
the  concepts  of  Integrated  Product  Teams  (IPT),  Integrated  Product  Development  (IPD), 
Integrated  Product  and  Process  Development  (IPPD)  and  concurrent  engineering.  In  this 
respect,  M&S  provides  the  ideal  mechanism  to  facilitate  all  four  elements. 

Specifically,  simultaneous  development  is  credited  with  the  following  benefits  : 
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1 .  Improving  the  quality  of  designs,  resulting  in  dramatic  reductions  of  engineering 
change  orders  (greater  than  50  percent)  in  early  production. 

2.  Reducing  product  development  cycle  time  by  as  much  as  40  to  60  percent  through 
concurrent,  rather  than  sequential,  design  of  product  and  processes. 

3.  Reducing  manufacturing  costs  by  as  much  as  30  to  40  percent  by  having 
multifunction  teams  integrate  product  and  process  designs. 

4.  Reducing  scrap  and  rework  by  as  much  as  75  percent  through  product  and  process 
design  optimization. 

A  substantial  portion  of  the  M&S  cost  benefits  arise  from  early  resolution  of  design  problems. 
Figure  229  clearly  illustrates  the  relatively  lower  cost  of  engineering  changes  as  well  as  their 
fewer  number  when  comparing  a  traditional,  serial  approach  to  development  with  a  concurrent 
engineering  approach  to  development. 

If  the  assessment  of  concurrent  engineering  is  not  attractive  enough  from  a  cost 
efficiency  perspective,  it  cements  its  credibility  with  the  observation  that  proper  implementation 
of  a  concurrent-engineering  approach  requires,  “early  consideration  of  a  product’s 
manufacturing  and  support  processes  while  shaping  the  user’s  [emphasis  added]  requirements” 
into  the  design.30  The  user’s  requirement  drives  the  military  acquisition  process,  and  it  plays  an 
even  larger  role  in  the  case  of  spiral  development.  A  collaborative  approach  to  product 
development  that  includes  the  user  is  far  more  likely  to  succeed  than  one  that  does  not  actively 
seek  user  input.  M&S  affords  the  user  an  opportunity  to  participate  directly  throughout  the 
entire  design  process  and  to  do  so  with  tools  that  exploit  his  subject  matter  expertise,  like 
reconfigurable,  man-in-the-loop  simulators. 

The  mental  picture  the  reader  should  have  at  this  point  is  that  of  a  soldier  subject-matter- 
expert  fighting  a  weapon  system  concept  on  the  virtual  battlefield  and  evaluating  all  aspects  of 
performance  from  man-machine  interface  to  combat  effectiveness.  Simultaneously,  engineers 
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from  all  design  disciplines  evaluate  the  ability  of  the  design  to  perform  under  the  loads  and 
stresses  imposed  by  the  soldier  in  the  loop.  All  designers,  the  soldier  included,  examine  a 
common  concept  of  the  product  from  their  own  unique  perspective,  using  tools  best  suited  for 
their  discipline,  and  participate  in  improving  the  design. 

Several  NASA  web  sites31, 32  offer  insight  into  the  various  design  perspectives  that 
should  be  considered  in  the  design  process.  These  are  summarized  in  Table  2. 


Concepualizability 

Evaluability 

Marketability 

Designability 

Prototypability 

Testability 

Producibility 

Deployability 

Operability 

Supportability 

Evolvability 

Retireability 

Manageability 

Value 

Competitive  Advantage 

Quality 

TABLE  2  NASA  "DESIGN  FOR  ..."  CONSIDERATIONS 


The  NASA  listing  is  by  no  means  all-inclusive  and  is  noteworthy  both  for  items  it  includes  and 
for  items  that  are  glaringly  absent.  Designers  of  Army  systems  must  consider  a  wide  range  of 
additional  perspectives.  A  suggested  list  is  shown  in  Table  3. 


Lethality 

Survivability 

Transportability 

Reliability 

Disposability 

Maintainability 

Assembly 

Disassembly 

The  Environment 

Safety 

Durability 

Affordability 

TABLE  3  ARMY  "DESIGN  FOR  ..."  CONSIDERATIONS 


This  author’s  list  is  not  comprehensive  either,  and  the  adage,  “Where  you  stand  depends 
on  where  you  sit,”  applies  should  one  try  to  prioritize  the  importance  of  these  design 
considerations.  In  an  age  of  specialization,  it  is  expected  that  experts  will  skillfully  argue  their 
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cases  from  their  own  unique  perspective.  However,  depth  comes  at  the  expense  of  breadth, 
and  whatever  the  experts  gain  in  their  own  field,  they  lose  in  appreciation  and  understanding  of 
others.  Specialization  increases  the  tendency  toward  parochialism  in  presenting  one’s  case.  It 
calls  to  mind  the  engineering  posters  that  were  prevalent  in  the  early  1980s  that  showed  a 
product  design  as  seen  by  the  various  contributing  disciplines.  For  example,  the  tank,  as  seen 
by  the  armaments  engineer,  showed  an  inconsequential  tracked  vehicle  sporting  a  comically 
large  main  gun.  The  tank,  as  seen  by  the  suspension  engineer,  depicted  all  aspects  of 
trafficability,  e.g.,  springs,  road  wheels,  etc.,  as  being  grossly  larger  than  all  other  aspects  of  the 
design.  Usually  such  posters  contained  a  final  picture,  that  as  seen  by  the  user,  which 
represented  a  well-thought-out  answer.  The  fact  that  these  posters  were  popular  at  all  suggests 
that  even  among  professionals,  there  is  the  danger  of  parochialism. 

M&S  serves  as  a  means  to  promote  communication  among  the  design  disciplines. 
Teaming  serves  as  a  filter  to  built-in  bias.  Properly  implemented,  IPTs  will  weigh  the  benefits  of 
competing  design  ideas  and  resolve  them  at  the  team  level  without  resorting  to  management 
direction.  Even  so,  there  is  a  danger  that  management  will  be  swamped  with  information  and 
trade-off  decisions.  However,  considering  that  the  alternative  is  “hiding  one’s  head  in  the  sand” 
and  relying  solely  upon  inputs  from  a  select  group,  more  knowledge  is  better.  The  opening 
epigraph  from  Frank  Watterson’s  now-retired  comic  strip  Calvin  and  Hobbes  makes  the  point 
clearly. 

COST  AND  SCHEDULE 

...  although  the  technical  problems  had  been  recognized  early  enough  to  prevent 
the  [Challenger  Space  Shuttle]  disaster,  meeting  flight  schedules  and  cutting 
costs  were  given  a  higher  priority  than  safety. 

— U.S.  House  of  Representatives  Committee  on  Science  and  Technology33 

Any  discussion  of  M&S  potential  would  be  incomplete  if  it  did  not  address  cost  and 
schedule,  and  much  can  be  learned  from  examining  NASA’s  experience.  NASA’s  efforts  in 
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M&S  are  often  highlighted  as  shining  success  stories  for  the  military  to  emulate.  They  deserve 
closer  review  especially  since  NASA  program  managers  face  the  same  cost,  schedule,  and 
performance  demands  as  do  military  program  managers.  Several  NASA  failures,  in  face  of 
these  demands,  have  called  into  question  the  validity  of  the  “faster,  better,  cheaper”  philosophy. 

When  the  space  shuttle  Challenger  exploded  shortly  after  lift-off,  and  the  whole  seven- 
person  crew  was  killed  in  front  of  a  national  television  audience,  both  the  President  and  the 
Congress  began  immediate  and  wholesale  investigations  to  determine  the  cause  and  to  effect 
measures  that  would  prevent  it  from  happening  again.  The  Rogers  Commission,  charged  by 
the  President  with  assessing  the  accident,  determined  that  the  incipient  physical  cause  of  the 
Challenger  disaster  was  "...  the  destruction  of  seals  [‘O’  rings]  that  are  intended  to  prevent  hot 
gases  from  leaking  ...  during  propellant  burn.” 34  The  report  describes  the  sequence  of  events 
resulting  from  the  decision  to  launch  the  Challenger  in  the  cold  early  morning  hours  of  January 
28, 1986.  In  the  near-freezing  temperatures,  the  ‘O’  rings  were  not  sufficiently  pliable  to  seal 
the  solid  rocket  booster  joint  and  prevent  the  escape  of  hot  gasses  after  launch.  The  resulting 
leak  of  flame  and  hot  gas  from  one  of  the  booster  rockets  weakened  the  strut  that  attached  the 
booster  to  the  external  fuel  tank.  Within  seconds,  the  strut  failed  and  the  booster  slammed  into 
the  shuttle  wing  and  the  external  tank  causing  the  explosion.  Engineers  had  previously 
surfaced  concerns  about  the  ‘O’  rings  and  also  highlighted  the  potential  danger  of  launching  in 
cold  weather  on  the  day  of  the  flight,  but  their  concerns  were  overruled  in  favor  of  meeting 
NASA’s  demanding  flight  schedule  — 15  missions  in  calendar  year  1986.  The  Rogers 
Commission  specifically  faulted  the  management  structure  of  both  the  prime  contractor,  Thiokol, 
and  that  of  NASA  for  not  allowing  critical  information  to  reach  the  right  people.35  Congress’ 
assessment  in  the  opening  epigraph  was  much  more  pointed  and  far  more  suggestive  of  the 
pressures  NASA  engineers  must  have  faced.  Overemphasis  on  cost  and  schedule  was  the 
ultimate  cause  of  the  explosion;  the  failed  seals  were  just  a  symptom. 
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The  climate  created  by  the  administration’s  demanding  cost  and  schedule  requirements 
is  similar  to  the  one  that  currently  exists  in  NASA  under  the  “faster,  better,  cheaper”  philosophy 
of  product  development.  Moreover,  the  effects  are  as  clear  today  as  they  were  at  the  time  of 
the  Challenger  disaster  because  NASA  suffered  the  loss  of  all  Mars’  spacecraft  in  the  last  year. 
The  physical  causes  of  the  failures  can  be  as  easily  explained  as  the  cause  of  the  Challenger 
explosion.  The  Mars  Climate  Orbiter  was  lost  because "...  contractor  Lockheed-Martin’s  Mars 
team  forgot  to  convert  from  English  to  metric  units  and  NASA  engineers  failed  to  catch  the 
error.”36  As  a  result,  the  orbiter  approached  Mars  at  the  wrong  altitude  and  was  lost  in  space. 
NASA  engineers  postulate  that  the  Mars  Polar  Lander  was  lost  because  of  a  single,  missing  line 
of  computer  code.37  Without  the  proper  coding,  the  Lander  mistakenly  “thought”  it  had  safely 
landed  on  the  surface  and  prematurely  shut  down  its  braking  thrusters  while  it  was  still  130  feet 
above  the  surface.  It  was  most  likely  destroyed  upon  impact.  The  fate  of  two  piggy-back 
probes  is  still  uncertain.  The  explanation  for  the  loss  of  the  Polar  Lander  is  only  an  engineering, 
best  guess.  Telemetry,  which  would  have  communicated  the  Lander’s  condition  during  descent 
into  the  Mars  atmosphere  and  would  have  conclusively  explained  the  cause,  was  eliminated  as 
a  cost-saving  measure  earlier  in  the  program. 

“Faster,  better,  cheaper”  will  most  likely  be  forever  seen  as  a  dictum  to  “do  more  with 
less.”  NASA  investigations  of  the  failures  concluded  that  the  Mars  projects  were  “underfunded, 
understaffed  and  over  stressed.”38  The  investigation  observed  that  managers,  against  their 
better  instincts,  may  have  failed  to  raise  their  concerns  for  fear  of  losing  ground  in  competition 
for  tight  funding.  NASA’s  top  scientist,  Edward  Weiler,  acknowledging  the  fundamental  flaw  of 
the  “faster,  better,  cheaper”  philospohy  stated,  “We’ve  found  the  boundary.” 39  Implicit  in  his 
assessment  is  that  the  effects  were  unacceptable  and  counterproductive  to  the  goal  of 
increased  efficiency. 

“Faster,  better,  cheaper”  is  often  heard  in  military  acquisition  circles,  but  almost  all 
program  mangers  privately  scoff  at  the  logic  behind  it.  Instead  they  adhere  to  the  ‘Management 
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101”  adage  concerning  cost,  schedule  and  performance,  i.e.,  “You  can  only  pick  two  of  the 
three.”  Figure  3  illustrates  the  inherent  interrelationship  of  cost,  schedule  and  performance  in 
terms  of  a  feasibility  space  or  trade  space.40  It  simplistically  illustrates  the  interdependence  of 
the  variables  and  clearly  shows  that  the  manager  only  has  the  freedom  to  control  two  of  the 
three  variables,  e.g.,  by  fixing  performance  and  schedule,  one  has  no  flexibility  in  varying  the 
cost.  The  trade  space  is  proposed  as  a  means  to  identify  options  in  the  development  process. 
Each  point  on  the  surface  represents  a  materiel  solution  which  can  be  evaluated  based  upon 
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performance  needs,  budget  constraints,  and  required  timeliness  of  delivery.  In  practice,  the 
trade  space  would  not  be  anywhere  near  so  smooth  nor  so  well  behaved  as  the  ellipsoid 
suggests;  it  would  most  likely  be  irregular  and  discontinuous.  Moreover,  it  would  be  a  significant 
exercise  to  define  the  whole  trade  space  for  any  reasonably  complex  system.  Still,  M&S  can  be 
effectively  used  to  populate  the  trade  space  with  at  least  some  of  the  options  so  that  leadership 
can  make  more-informed  decisions. 
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If  NASA’s  successes  motivated  the  Army  to  strive  for  similar  gains,  then  NASA’s  failures 
should  similarly  motivate  the  Army  to  at  least  re-examine  its  policies  and  its  preoccupation  with 
near-term  cost  savings  in  defining  expectations  for  M&S.  Figure  2  clearly  demonstrates  the  cost 
efficiencies  associated  with  early  detection  and  correction  of  design  deficiencies.  However, 
design  changes  do  require  resources,  both  time  and  funding,  to  implement.  The  benefits,  which 
are  realized  from  the  design  change,  are  categorized  as  cost  avoidance,  while  the  expense 
incurred  for  the  change  is  a  true  cost.  To  put  things  in  perspective,  consider  the  engineering 
change  implemented  by  the  Family  of  Medium  Tactical  Vehicles  (FMTV)  program  office  to 
correct  a  design  flaw  that  caused  excessive  vibration  in  the  drivetrain  at  highway  speeds.  Since 
the  engineering  change  was  implemented  during  production  and  after  some  vehicles  had  been 
fielded,  the  FMTV  program  incurred  costs  that  were  at  least  an  order  of  magnitude  higher  than 
would  have  been  incurred  had  the  flaw  been  detected  and  fixed  before  production  began. 
However,  if  the  flaw  had  been  detected  earlier  in  the  program,  the  resulting  expense  and  time  to 
implement  the  fix  might  have  been  categorized  as  cost  growth  and  schedule  slip,  the  two  things 
NASA  managers  feared  most  in  the  Mars  program.  Cost  and  schedule  must  be  viewed  from  the 
perspective  of  the  overall,  system  life-cycle  if  models  and  simulations  are  ever  to  truly  change 
the  acquisition  process. 

RECOMMENDATIONS 

This  author  recognizes  that  any  discussion  of  the  use  of  M&S  in  the  acquisition  process 
is  moot.  Models  ancf  simulations  are  inextricably  linked  to  the  engineering,  science  and 
analytical  professions  that  are  responsible  for  developing  and  fielding  military  hardware.  In  spite 
of  their  limitations,  models  and  simulations  are  powerful  tools  for  improving  the  decision-making 
process.  The  only  open  question  is,  “Can  these  tools  be  more  effectively  exploited  in  the 
acquisition  process?”  With  that  goal  in  mind,  I  propose  the  following  recommendations: 
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1 .  Focus  modeling  and  simulation  efforts  on  Army  applications  that  specifically  advance 
the  CSA’s  vision  for  transforming  the  Army.  The  Program  Manager,  Future  Combat 
System  (FCS)  should  establish  an  M&S  milestone  to  evaluate  design  concepts. 
Concurrently,  TRADOC  should  develop  the  scenarios.  It  should  be  noted  that  the 
FCS  solicitation41  addresses  many  of  the  points  raised  in  this  paper,  i.e.,  simulation- 
based  approach,  teaming,  etc.  The  benefits  of  establishing  a  simulation  milestone  to 
the  FCS  schedule  would  be  to  focus  many  related  efforts,  e.g.,  simulation 
interoperability,  subsystem  modeling,  etc,  and  to  provide  a  needed  sense  of  urgency 
in  the  M&S  community. 

2.  Adopt  a  vision  statement  that  evokes  the  mental  picture  suggested  in  Figure  1 ,  and 
encourage  program  managers  to  contribute  to  the  vision  by  extending  their  program- 
specific  M&S  efforts  for  more  general  use.  The  bulk  of  M&S  efforts  are  performed  in 
individual  program  offices;  with  proportionately  little  extra  resources,  these 
simulations  can  be  made  to  be  more  general  purpose,  especially  if  a  program 
manger  begins  a  simulation  effort  with  wider  use  in  mind.  To  do  that,  however,  he 
must  have  a  clear  picture  of  what  is  expected.  Current  vision  statements  leave  too 
much  room  to  conclude  that  it’s  someone  else’s  responsibility. 

3.  Incrementally  and  formally  incorporate  mature  modeling  techniques  (low,  hanging 
fruit)  into  the  acquisition  process  beginning  with  the  transportability  and  deployability- 
analysis  tools  maintained  by  the  Military  T raffic  Management  Command, 
Transportation  Engineering  Agency.  These  tools  are  low  risk,  mature,  proven,  and 
are  directly  applicable  to  the  Chief’s  transformation  vision.  Benefits  would  be  twofold: 
first,  formal  recognition  of  this  capability  would  reduce  the  likelihood  resources  would 
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be  spent  elsewhere,  duplicating  this  capability, 
and  second,  these  tools  could  “proof”  the 
process  by  which  other  models  are 
incorporated  into  the  acquisition  process.  The 
learning  curve  has  to  start  somewhere. 

4.  Implement  and  accept  a  peer-review 

alternative  to  the  current  VV&A  process  for 


FIGURE  4 

TRANSPORTABILITY  authenticating  M&S  results.  Peer  review  is 

ANALYSIS 

a  natural  outgrowth  of  the  collaborative 
engineering  approach  using  distributed  simulation  and  modeling  tools.  Place  the 
burden  for  explanation  and  support  of  results  with  the  modeler,  and  recognize  that 
the  primary  benefit  of  M&S  is  risk  reduction  (not  risk  elimination)  resulting  from 
extensive  collaboration  among  experts  from  all  aspect  of  the  acquisition  process. 


5.  Develop  and  track  appropriate  metrics.  Begin  with  life-cycle  cost.  The  NASA 

experience  has  demonstrated  the  fallacy  of  “faster,  better,  cheaper,”  and  its  focus  on 
near-term  savings. 


CONCLUSION 

...  convergence  has  meant  two  things:  the  coming  together  of  the  computer, 
consumer  electronics,  and  telecommunications  industries  and  the  merging  of 
gadgets  such  as  the  PC,  TV  and  telephone.  But  this  is  only  the  start.  Combine 
digital  technology  with  advanced  software,  smaller  and  more  powerful 
microprocessors,  and  exponential  growth  in  fiber  and  wireless  bandwidth,  and 
you  get  something  far  more  useful  —  seamless,  universal  connectivity.  ...  Call  it 
“virtual”  convergence  —  everything  you  want  in  one  place,  but  that  place  is 
wherever  you  want  it  to  be. 

—Bill  Gates42 
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Simulation  Growth 


From  his  position  as  President 
for  Research  and  Development  at  Walt 
Disney  Imagineering,  Mr.  Ferren  has  a 
unique  perspective  on  evolving 

Time 

technologies.  He  vividly  described  the 

FIGURE  5  SIMULATION  GROWTH  potentia|  for  M&s  to  modem|ze  ,he 

acquisition  process  in  terms  of  the  exponential  curve  in  Figure  5.43  He  then  asked  the  audience 
attending  the  most  recent  Simulation  and  Modeling  for  Acquisition,  Requirements,  and  Training 
conference  where  they  believed  the  Army  was  on  the  growth  continuum.  Most  of  the  audience, 
having  witnessed  the  many  changes  precipitated  by  M&S  over  the  last  decade,  pointed  to  a 
point  on  the  “knee”  of  the  curve,  i.e.,  Point  A.  Mr.  Ferren  disagreed.  He  emphasized  that  we 
have  barely  “left  the  gate”  in  the  computer  revolution  and  that  the  audience  had  better  adjust  its 
reference  to  a  point  nearer  to  Point  B  because  the  greatest  rate  of  change  is  yet  to  come. 

At  the  same  conference,  a  panel  of  General  Officers  was  asked  why,  after  a  decade  of 
pursuing  the  vision  of  a  simulation-based  acquisition  process,  are  we  not  “there”  yet.  MG  John 
Caldwell,  currently  the  Commanding  General  of  the  U.S.  Army  Tank,  Automotive  and 
Armaments  Command,  attributed  it  to  a  lack  of  focus.44  He  likened  the  situation  to  a  group  of 
talented  engineers,  who  in  the  absence  of  specific  direction,  tend  to  work  only  on  problems  that 
interest  them.  He  further  commented  that  the  answer  is  to  provide  them  direction,  and  that  in 
the  case  of  M&S,  it  should  be  to  focus  efforts  in  support  of  the  Chief’s  Transformation  Vision. 

The  opening  epigraph  describes  the  concept  of  convergence  and  the  potential  that 
results  from  combining  technologies.  Those  pursuing  the  M&S  vision  stand  at  a  crossroads 
where  technology,  need,  and  interest  converge.  M&S  tools  have  improved  dramatically  over  the 
last  decade.  The  need  to  transform  the  Army  into  a  more  deployable  force  has  become  the 
Army’s  top  priority  and  represents  an  ideal  opportunity  to  use  M&S  to  advantage.  Interest  in 
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applying  ever  more  capable  M&S  tools  to  revamp  the  acquisition  process  is  as  high  as  it  has 
ever  been.  The  Army  has  the  means,  the  desire  and  the  motivation  to  succeed.  Will  the 
question,  “Why  aren’t  we  there  yet?”  be  asked  again  a  decade  from  now? 
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